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Abstract. The conjugacy search problem in a group G is the problem 
of recovering an s £ G from given g £ G and h = x~^gx. The alleged 
computational hardness of this problem in some groups was used in sev- 
eral recently suggested public key exchange protocols, including the one 
due to Anshel, Anshel, and Goldfeld, and the one due to Ko, Lee et 
al. Sibert, Dehornoy, and Girault used this problem in their authenti- 
cation scheme, which was inspired by the Fiat-Shamir scheme involving 
repeating several times a three-pass challenge-response step. 
In this paper, we offer an authentication scheme whose security is based 
on the apparent hardness of the twisted conjugacy search problem which 
is: given a pair of endomorphisms (i.e., homomorphisms into itself) ^, ?/> 
of a group G and a pair of elements w,t € G, find an element s £ G 
such that t = ijj{s~^)w(p{s) provided at least one such s exists. This 
problem appears to be very non-trivial even for free groups. We offer 
here another platform, namely, the semigroup of all 2 x 2 matrices over 
truncated one- variable polynomials over F2, the field of two elements, 
with transposition used instead of inversion in the equality above. 



1 Introduction 

One of the most obvious ramifications of the discrete logarithm problem in the 
noncommutative situation is the conjugacy search problem: 

Given a group G and two conjugate elements g,h £ G, find a particular 
element x £ G such that x^^gx ~ h. 

This problem always has a recursive solution because one can recursively 
enumerate all conjugates of a given element, but this kind of solution can be 
extremely inefficient. Specific groups may or may not admit more efficient so- 
lutions, so the choice of the platform group is of paramount importance for 
security of a cryptographic primitive based on the conjugacy search problem. A 
great deal of research was (and still is) concerned with the complexity of this 
problem in braid groups because there were several proposals, including the one 
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by Anshel, Anshel, and Goldfeld [1], and the one by Ko, Lee at al. [11] on using 
the alleged computational hardness of this problem in braid groups to build a key 
exchange protocol. Also, Sibert, Dehornoy, and Girault [15] used this problem 
in their authentication scheme, which was inspired by the Fiat-Shamir scheme 
involving repeating several times a three-pass challenge-response step. At the 
time of this writing, no deterministic polynomial-time algorithm for solving the 
conjugacy search problem in braid groups has been reported yet; see [3] and [4] 
for recent progress in this direction. However, several heuristic algorithms, in 
particular so-called "length based attacks" , were shown to have very high suc- 
cess rates, see e.g. [7], [8], [10], [12], [13]. This shows that one has to be really 
careful when choosing the platform (semi)group to try to avoid length based or 
similar attacks. One way to achieve this goal is, informally speaking, to have "a 
lot of commutativity" inside otherwise non-commutative (semi) group; see [13] 
for a more detailed discussion. 

In this paper, we propose an authentication scheme whose security is based 
on the apparent hardness of the (double) twisted conjugacy search problem which 
is: 

given a pair of endomorphisms (i.e., homomorphisms into itself) ip,ip of 
a group G and a pair of elements w,t G G, find an element s € G such 
that t = ■4'{s~^)wip{s) provided at least one such s exists. 

This problem, to the best of our knowledge, has not been considered in group 
theory before, and neither was its decision version: given (p,'ip £ End{G), w, t € 
G, find out whether or not there is an element s € G such that t = ip{s~^)wip{s). 
However, the following special case of this problem (called the twisted conjugacy 
problem) has recently attracted a lot of interest among group theorists: 

given ip G End{G), w,t G G, find out whether or not there is an element 
s £ G such that t = s~^w(f{s). 

This problem is very non-trivial even for free groups; see [5] for an aston- 
ishing solution in the special case where (p is an automorphism of a free group. 
To the best of our knowledge, this decision problem is open for free groups if <fi 
is an arbitrary endomorphism. Another class of groups where the twisted con- 
jugacy problem was considered is the class of polycyclic-by-finite groups [16]. 
Again, the problem was solved for these groups in the special case where ip is an 
automorphism. 

The conjugacy problem is a special case of the twisted conjugacy problem, 
where <p is the identity map. Now a natural question is: what is the advantage of 
the more general (double) twisted conjugacy search problem over the conjugacy 
search problem in the context of an authentication scheme? The answer is: if the 
platform (semi) group G has "a lot" of endomorphisms, then Alice (the prover), 
who selects ip, tp, w, and s, has an opportunity to select them in such a way that 
there are a lot of cancelations between tp{s),w, and ip{s), thus rendering length 
based attacks ineffective. 



In this paper, we use the semigroup of all 2 x 2 matrices over truncated 
one- variable polynomials over F2, the field of two elements, as the platform. It 
may seem that the platform necessarily has to be a group since one should at 
least have the element s (see above) invertible. However, as we will see in the 
next section, we do not really need the invertibility to make our authentication 
protocol work; what we need is just some antihomomorphism of G into itself, i.e., 
a map * : G ^ G such that (ab)* = b*a* for any a,b G G. Every group has such 
an antihomomorphism; it takes every element to its inverse. Every semigroup 
of square matrices has such an antihomomorphism, too; it takes every matrix 
to its transpose. Some (semi)groups have other special antihomomorphisms; for 
example, any free (semi)group has an antihomomorphism that rewrites every 
element "backwards", i.e., right-to- left. Here we prefer to focus on semigroups 
of matrices (over commutative rings) since we believe that these have several 
features making them fit to be platforms of various cryptographic protocols, see 
[14] for a more detailed discussion. 

2 The protocol 

In this section, we give a description of a single round of our authentication pro- 
tocol. As with the original Fiat-Shamir scheme, this protocol has to be repeated 
k times if one wants to reduce the probability of successful forgery to ^ . 

Here Alice is the prover and Bob the verifier. Let G be the platform semi- 
group, and * an antihomomorphism of G, i.e., (ab)* = b*a*. 

1. Alice's public key is a pair of endomorphisms (p, tp of the group G and two 
elements w,t e G, such that t = tp{s*)w(p{s), where s e G is her private key. 

2. To begin authentication, Alice selects an element r € G and sends the ele- 
ment u = 'il){r*)t(fi{r), called the commitment, to Bob. 

3. Bob chooses a random bit c and sends it to Alice. 

— If c = 0, then Alice sends v — r to Bob and Bob checks if the equality 
u = tp{v*)t(fi{v) is satisfied. If it is, then Bob accepts the authentication. 

— If c = 1, then Alice sends v = sr to Bob and Bob checks if the equality 
u = ip{v*)iu(p{v) is satisfied. If it is, then Bob accepts the authentication. 

Let us check now that everything works the way we want it to work. 

— If c = 0, then v — r, so tp{v*)t(p{v) = ip{r*)t<f{r) = u. 

— If c = 1, then V = sr, so ■tp{v*)'Wip{v) = ip{{sr)*)'Wip{sr) = tjj{r* s*)'Wip{s)ip{r) = 
ip{r*)tp{s*)wip{s)ip{r) = u. 

3 The platform and parameters 

Our suggested platform semigroup G is the semigroup of all 2 x 2 matrices over 
truncated one- variable polynomials over F2, the field of two elements. Truncated 
(more precisely, A''-truncated) one-variable polynomials over F2 are expressions 



of the form a^x*, where at are elements of F2, and x is a variable. In 

0<i<JV-l 

other words, Af-truncated polynomials are elements of the factor algebra of the 
algebra F2[a;] of one- variable polynomials over F2 by the ideal generated by . 

Our semigroup G has a lot of endomorphisms induced by endomorphisms of 

the algebra of truncated polynomials. In fact, any map of the form x —>■ p{x), 
where p{x) is a truncated polynomial with zero constant term, can be extended 
to an endomorphism (j)p of the algebra of truncated polynomials. Indeed, it is 
sufficient to show that (pp{x^) = {p{x))^ belongs to the ideal generated by x^, 
which is obviously the case if p{x) has zero constant term. Then, since (j)p is both 
an additive and a multiplicative homomorphism, it extends to an endomorphism 
of the semigroup of all 2 x 2 matrices over truncated one-variable polynomials 
in the natural way. 

If we now let the antihomomorphism * from the description of the protocol 

in our Section 2 to be the matrix transposition, wc have everything set up for 
an authentication scheme using the semigroup G as the platform. 

Now we have to specify parameters involved in our scheme. The parameter 
N determines the size of the key space. If N is on the order of 300, then there are 
2-300 polynomials of degree < N over F2, so there arc 2^^°" 2x2 matrices over 
A'^-truncated polynomials, i.e., the size of the private key space is 2^^°", which 
is large enough. 

At the same time, computations with (truncated) polynomials over F2 are 
very efficient (see e.g. [2], [6], or [9] for details). In particular. 



— Addition of two polynomials of degree N can be performed in 0{N) time. 

— Multiplication of two polynomials of degree A'' can be performed in 0{N log2 N) 
time. 

— Computing composition p{q{x)) mod x^ of two polynomial of degree N can 
be performed in 0((iVlog2 iV)2) time (see e.g. [6, p. 51]). 

Since those are the only operations used in our protocol, the time complexity 
of executing a single round of the protocol is 0((iVlog2 N)^). 

The size of public key space is large, too. One public key is, again, a 2 x 2 
matrix over A^-truncated polynomials, and two other public keys are endomor- 
phisms of the form x p{x). where p{x) is an A^-truncatcd polynomial with 
zero constant term. Thus, the number of different endomorphisms in this context 
is on the order of 2'^"°, hence the number of different pairs of endomorphisms is 
on the order of 2^°". 

We also have to say a few words about how a private key s G G is selected. 
We suggest that all entries of the matrix s have non-zero constant term; other 
coefficients of the entries can be selected randomly, i.e., "0" and "1" arc selected 
with probability ^ each. Non-zero constant terms are useful here to ensure that 
there are sufficiently many non-zero terms in the final product t = ip{s*)'Wip{s) . 



4 Cryptanalysis 



As we have pointed out in the previous section, the key space with suggested 
parameters is quite large, so that a "brute force" attack by exhausting the key 
space is not feasible. 

The next natural attack that comes to mind is attempting to solve a system 
of equations over F2 that arises from equating coefficients at the same powers 
of X on both sides of the equation t = tp{s*)w(p{s). Recall that in this equation 
t, w, ip, and ip are known, whereas s is unknown. 

More specifically, our experiments emulating this attack were designed as 
follows. The entries of the private matrix s were generated as polynomials of 
degree A^ — 1, with iV = 100 (which is much smaller than the suggested N = 300), 
with randomly selected binary coefhcients, except that the constant term in 
all polynomials was 1. Then, the endomorphisms ip and ip were of the form 
X — > Pi{x), where Pi{x) are polynomials of degree TV — 1, with N = 150, with 
randomly selected binary coefficients, except that the constant term in both of 
them was 0. Finally, the entries of the public matrix w were generated, again, 
as polynomials of degree N ~ 1, with N = 100, with randomly selected binary 
coefficients, except that the constant term in all polynomials was 1. 

The attack itself then proceeds as follows. The matrix equation t — tp{s*)w(p{s) 
is converted to a system of 4N polynomial equations {N for each entry of a 2 x 2 
matrix) over F2. The unknowns in this system are coefficients of the polynomials 
of degree N — 1 that are the entries of the private matrix s. Then, starting with 
the constant term and going up, we equate coefficients at the same powers of x 
on both sides of each equation. After that, again starting with the coefficients at 
the constant term and going up, wc find all possible solutions of each equation, 
one at a time. Thus we are getting a "trcx;" of solutions because some of the un- 
knowns that occur in coefficients at lower powers of x also occur in coefficients at 
higher powers of x. If this tree does not grow too fast, then there is a chance that 
we can get all the way to the coefficients at highest power of x, thereby finding a 
solution of the system. This solution may not necessarily yield the same matrix 
s that was selected by Alice, but it is sufficient for forgery anyway. 

We have run over 1000 experiments of this kind (which took about two 
weeks), allowing the solution tree to grow up to the width of 16384, i.e., allowing 
to go over at most 16384 solutions of each equation when proceeding to a higher 
power of X. Each experiment ran on a personal computer with Pentium 2Ghz dual 
core processor. The success rate of the described attack with these parameters 
was 0%. 

5 Conclusions 

We have introduced: 

1. An authentication scheme based on the (double) twisted conjugacy problem, 
a new problem, which is allegedly hard in some (semi)groups. 



2. A new platform semigroup, namely the semigroup of all 2 x 2 matrices over 
truncated one- variable polynomials over F2. Computation in this semigroup 
is very efficient and, at the same time, the non-commutative structure of this 
semigroup provides for security at least against obvious attacks. 

We point out here one important advantage of using the (double) twisted 
conjugacy problem over using a more "traditional" conjugacy search problem as 
far as (semi)groups of matrices are concerned. The conjugacy search problem 
admits a linear algebra attack upon rewriting the equation x~^gx = h as gx = 
xh; the latter translates into a system of linear equations with unknowns, 
where n is the size of the matrices involved, and the unknowns are the entries 
of the matrix x. Of course, if the entries come not from a field but from a 
more general ring, such a system of linear equations does not necessarily admit 
a straightforward solution, but methods emulating standard techniques (like 
Gauss elimination) usually have a pretty good success rate anyway. For the 
twisted conjugacy problem, however, there is no reduction to a system of linear 
equations. 

We have considered an attack based on reducing the twisted conjugacy prob- 
lem to a system of polynomial equations over F2 , but this attack becomes com- 
putationally infeasible even with a much smaller crucial parameter (which is the 
maximum degree of the polynomials involved) than the one we suggest in this 
paper. 
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